Instruction profiling using multiple metrics

ABSTRACT

A system and method for collecting a plurality of metrics during a single run of a computer program. The mechanism of the present invention initializes a plurality of counters to count events associated with metrics of interest. The mechanism of the present invention then counts the occurrence of events associated with metrics of interest during a single execution of a computer program. Responsive to a determination that a counter in a plurality of counters has generated an interrupt, the interrupt is rerouted to an interrupt handler, wherein the interrupt handler generates trace records comprising trace information corresponding to the interrupt. The mechanism of the present invention then generates profiles for the trace records, wherein the profiles differentiate the trace records based on the metric type associated with each trace record.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to commonly assigned and co-pending U.S.patent application Ser. No. ______ (Attorney Docket No.AUS9200400693US1) entitled “System and Method for Collecting a Pluralityof Metrics in a Single Profiling Run of Computer Code” filed even dateherewith, and which is hereby incorporated by reference.

GOVERNMENT RIGHTS

This invention was made with Government support under NBCH30390004,PERCS project. THE GOVERNMENT HAS CERTAIN RIGHTS IN THIS INVENTION.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention is generally directed to an improved dataprocessing system. More specifically, the present invention is directedto a system and method for performing profiling of the execution of aportion of computer code to allow for collection of a plurality ofmetrics during a single profiling run

2. Description of Related Art

In analyzing and enhancing performance of a data processing system andthe applications executing within the data processing system, it ishelpful to know which software modules within a data processing systemare using system resources. Effective management and enhancement of dataprocessing systems requires knowing how and when various systemresources are being used. Performance tools are used to monitor andexamine a data processing system to determine resource consumption asvarious software applications are executing within the data processingsystem. For example, a performance tool may identify the most frequentlyexecuted modules and instructions in a data processing system, or mayidentify those modules which allocate the largest amount of memory orperform the most I/O requests. Hardware performance tools may be builtinto the system or added at a later point in time.

One known software performance tool is a trace tool. A trace tool mayuse more than one technique to provide trace information that indicatesexecution flows for an executing program. One technique keeps track ofparticular sequences of instructions by logging certain events as theyoccur, so-called event-based profiling technique. For example, a tracetool may log every entry into, and every exit from, a module,subroutine, method, function, or system component. Alternately, a tracetool may log the requester and the amounts of memory allocated for eachmemory allocation request. Typically, a time-stamped record is producedfor each such event. Corresponding pairs of records similar toentry-exit records also trace execution of arbitrary code segments,starting and completing I/O or data transmission, and for many otherevents of interest.

In order to improve performance of code generated by various families ofcomputers, it is often necessary to determine where time is being spentby the processor in executing code, such efforts being commonly known inthe computer processing arts as locating “hot spots.” Ideally, one wouldlike to isolate such hot spots at the instruction and/or source line ofcode level in order to focus attention on areas which might benefit mostfrom improvements to the code.

Another trace technique involves periodically sampling a program'sexecution flows to identify certain locations in the program in whichthe program appears to spend large amounts of time. This technique isbased on the idea of periodically interrupting the application or dataprocessing system execution at regular intervals, so-called sample-basedprofiling. At each interruption, this trace technique recordsinformation for a predetermined length of time or for a predeterminednumber of events of interest. For example, the program counter of thecurrently executing thread, which is an executable portion of the largerprogram being profiled, may be recorded at each interval. This analysismay allow for resolving the recorded values against a load map andsymbol table information for the data processing system atpost-processing time and for obtaining a profile of where the time isbeing spent.

Typically, known performance tools are capable of monitoring andanalyzing the performance of a data processing system and theapplications executing within the data processing system with regard toa single metric, e.g., CPU cycles, number of instructions, etc. That is,each metric of interest requires a single run of an application orprogram being traced. Thus, in a first run of the program, theperformance tool may measure a first metric. In order to obtainperformance information for an application with regard to anothermetric, the application must be run again with a performance tool thatis capable of monitoring the application with regard to this othermetric.

When applied to simulations, monitoring performance data and tracing theexecution of such simulations may take large amounts of time. In fact,some simulations may take multiple days to complete. Typically, it isdesired that performance data with regard to a plurality of differentmetrics be obtained from such simulations. However, with known systems,such performance data collection, tracing, and post processing toproduce sampled base reports or reports by subroutine roll ups can onlybe performed with regard to single metrics per run of the simulation. Asa result, multiple runs of a simulation are necessary to obtain all ofthe desired reports. This greatly increases the time and expense inobtaining the reports needed for performance analysis, verification, andthe like.

Thus, it would be beneficial to have a system and method for collectinga plurality of metrics in a single profiling run of a computer code.

SUMMARY OF THE INVENTION

The present invention provides a system and method for collecting aplurality of metrics during a single run of a computer program. Themechanism of the present invention initializes a plurality of countersto count events associated with metrics of interest. The mechanism ofthe present invention then counts the occurrence of events associatedwith metrics of interest during a single execution of a computerprogram. Responsive to a determination that a counter in a plurality ofcounters has generated an interrupt, the interrupt is rerouted to aninterrupt handler, wherein the interrupt handler generates trace recordscomprising trace information corresponding to the interrupt. Themechanism of the present invention then generates profiles for the tracerecords, wherein the profiles differentiate the trace records based onthe metric type associated with each trace record.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives and advantages thereof, willbest be understood by reference to the following detailed description ofan illustrative embodiment when read in conjunction with theaccompanying drawings, wherein:

FIG. 1 is an exemplary pictorial representation of a distributed dataprocessing system in which the present invention may be implemented;

FIG. 2 is an exemplary block diagram of a server data processing systemin which aspects of the present invention may be implemented;

FIG. 3 is an exemplary block diagram of a stand-alone or client dataprocessing system in which aspects of the present invention may beimplemented;

FIG. 4 is an exemplary block diagram depicting components used toperform performance traces of processes in a data processing system;

FIG. 5 is a diagram depicting various phases in performing a performancetrace of the workload running on a system;

FIG. 6 is a block diagram of a processor system for processinginformation according to the preferred embodiment;

FIG. 7 is a diagram illustrating primary operational componentsaccording to one exemplary embodiment of the present invention;

FIG. 8A illustrates an example of the trace records generated forexemplary embodiments of the present invention in which the trace recordis generated in response to a counter overflow generating an interrupt;

FIG. 8B illustrates an exemplary trace record for an embodiment of thepresent invention in which trace records are generated in response to atimer interrupt;

FIG. 8C illustrates an exemplary trace record for an embodiment of thepresent invention in which trace records are generated for every branchtaken in an application under trace;

FIG. 9 is a flowchart outlining an exemplary operation of the presentinvention in which trace records are generated in response to a countergenerated interrupt;

FIG. 10 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention in which counter values are set totheir capacity in response to a timer interrupt; and

FIG. 11 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when generating trace records foreach counter at every branch taken in an application under trace.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention provides trace tools with functionality for usinga plurality of counters to count events associated with a plurality ofmetrics during a single run of a computer program. In one exemplaryembodiment, the counters generate interrupts when a maximum value of thecounter is reached. The interrupts cause the trace tools to generatetrace records identifying the event type associated with the countersthat generate the interrupts. Thereafter, a post-processing mechanismmay sort the trace records into sets of trace records based on the eventtype to thereby generate separate profiles for each metric of interest.

In another embodiment, the mechanism of the present invention uses atimer to reset the values of the counters to a maximum value when thetimer's time interval is reached. By setting the counters to theirmaximum value, the occurrence of a next event of the type counted by thecounters causes the counters to generate an interrupt. An interrupthandler handles the interrupt and then generates a trace recordidentifying the event type corresponding to the counter that generatedthe interrupt.

In a further embodiment, the mechanism of the present inventiongenerates trace records every time a branch is taken. For a branchtaken, the trace records are those only used for instruction traceprocessing. For illustrative purposes, the trace tool generates a singletrace record for each branch taken. This trace record contains all ofthe metrics for that branch taken. Many different variations could beimplemented, for example, the trace tool may issue one or more separatetrace record(s) for only those metrics that changed or the trace toolmay record the full value of the metric instead of the changes. In someembodiments, the required information may be compressed, for example,both the from and to branch addresses may not be required if the numberof instructions or distance from the last branch is recorded instead.Each trace record allows for the identification of the changed countersand the amount of the change since a last observation of the value ofeach of the changed counters. Post-processing may use this informationto separate out trace records into sets of trace records based on eventtype and to generate separate callstack trees for each metric based onthe separate sets of trace records.

The present invention may be implemented in a distributed dataprocessing environment or in a single data processing device. Therefore,the following FIGS. 1-3 are provided as exemplary environments in whichaspects of the present invention may be implemented. FIGS. 1-3 are onlyexemplary and are not intended to state or imply any limitation withregard to the types of data processing environments in which the presentinvention may be implemented. Many modifications to the exemplaryenvironments depicted may be made without departing from the spirit andscope of the present invention.

With reference now to the figures, FIG. 1 depicts a pictorialrepresentation of a network of data processing systems in which thepresent invention may be implemented. Network data processing system 100is a network of computers in which the present invention may beimplemented. Network data processing system 100 contains a network 102,which is the medium used to provide communications links between variousdevices and computers connected together within network data processingsystem 100. Network 102 may include connections, such as wire, wirelesscommunication links, or fiber optic cables.

In the depicted example, server 104 is connected to network 102 alongwith storage unit 106. In addition, clients 108, 110, and 112 areconnected to network 102. These clients 108, 110, and 112 may be, forexample, personal computers or network computers. In the depictedexample, server 104 provides data, such as boot files, operating systemimages, and applications to clients 108-112. Clients 108, 110, and 112are clients to server 104. Network data processing system 100 mayinclude additional servers, clients, and other devices not shown. In thedepicted example, network data processing system 100 is the Internetwith network 102 representing a worldwide collection of networks andgateways that use the Transmission Control Protocol/Internet Protocol(TCP/IP) suite of protocols to communicate with one another. At theheart of the Internet is a backbone of high-speed data communicationlines between major nodes or host computers, consisting of thousands ofcommercial, government, educational and other computer systems thatroute data and messages. Of course, network data processing system 100also may be implemented as a number of different types of networks, suchas for example, an intranet, a local area network (LAN), or a wide areanetwork (WAN). FIG. 1 is intended as an example, and not as anarchitectural limitation for the present invention.

Referring to FIG. 2, a block diagram of a data processing system thatmay be implemented as a server, such as server 104 in FIG. 1, isdepicted in accordance with a preferred embodiment of the presentinvention. Data processing system 200 may be a symmetric multiprocessor(SMP) system including a plurality of processors 202 and 204 connectedto system bus 206. Alternatively, a single processor system may beemployed. Also connected to system bus 206 is memory controller/cache208, which provides an interface to local memory 209. I/O Bus Bridge 210is connected to system bus 206 and provides an interface to I/O bus 212.Memory controller/cache 208 and I/O Bus Bridge 210 may be integrated asdepicted.

Peripheral component interconnect (PCI) bus bridge 214 connected to I/Obus 212 provides an interface to PCI local bus 216. A number of modemsmay be connected to PCI local bus 216. Typical PCI bus implementationswill support four PCI expansion slots or add-in connectors.Communications links to clients 108-112 in FIG. 1 may be providedthrough modem 218 and network adapter 220 connected to PCI local bus 216through add-in connectors.

Additional PCI bus bridges 222 and 224 provide interfaces for additionalPCI local buses 226 and 228, from which additional modems or networkadapters may be supported. In this manner, data processing system 200allows connections to multiple network computers. A memory-mappedgraphics adapter 230 and hard disk 232 may also be connected to I/O bus212 as depicted, either directly or indirectly.

Those of ordinary skill in the art will appreciate that the hardwaredepicted in FIG. 2 may vary. For example, other peripheral devices, suchas optical disk drives and the like, also may be used in addition to orin place of the hardware depicted. The depicted example is not meant toimply architectural limitations with respect to the present invention.

The data processing system depicted in FIG. 2 may be, for example, anIBM eServer pSeries system, a product of International Business MachinesCorporation in Armonk, N.Y., running the Advanced Interactive Executive(AIX) operating system or LINUX operating system.

With reference now to FIG. 3, a block diagram illustrating a dataprocessing system is depicted in which the present invention may beimplemented. Data processing system 300 is an example of a clientcomputer. Data processing system 300 employs a peripheral componentinterconnect (PCI) local bus architecture. Although the depicted exampleemploys a PCI bus, other bus architectures such as Accelerated GraphicsPort (AGP) and Industry Standard Architecture (ISA) may be used.Processor 302 and main memory 304 are connected to PCI local bus 306through PCI Bridge 308. PCI Bridge 308 also may include an integratedmemory controller and cache memory for processor 302. Additionalconnections to PCI local bus 306 may be made through direct componentinterconnection or through add-in boards. In the depicted example, localarea network (LAN) adapter 310, small computer system interface (SCSI)host bus adapter 312, and expansion bus interface 314 are connected toPCI local bus 306 by direct component connection. In contrast, audioadapter 316, graphics adapter 318, and audio/video adapter 319 areconnected to PCI local bus 306 by add-in boards inserted into expansionslots. Expansion bus interface 314 provides a connection for a keyboardand mouse adapter 320, modem 322, and additional memory 324. SCSI hostbus adapter 312 provides a connection for hard disk drive 326, tapedrive 328, and CD-ROM drive 330. Typical PCI local bus implementationswill support three or four PCI expansion slots or add-in connectors.

An operating system runs on processor 302 and is used to coordinate andprovide control of various components within data processing system 300in FIG. 3. The operating system may be a commercially availableoperating system, such as Windows XP, which is available from MicrosoftCorporation. An object oriented programming system such as Java may runin conjunction with the operating system and provide calls to theoperating system from Java programs or applications executing on dataprocessing system 300. “Java” is a trademark of Sun Microsystems, Inc.Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as hard disk drive 326, and may be loaded into main memory 304 forexecution by processor 302.

Those of ordinary skill in the art will appreciate that the hardware inFIG. 3 may vary depending on the implementation. Other internal hardwareor peripheral devices, such as flash read-only memory (ROM), equivalentnonvolatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIG. 3. Also, theprocesses of the present invention may be applied to a multiprocessordata processing system.

As another example, data processing system 300 may be a stand-alonesystem configured to be bootable without relying on some type of networkcommunication interfaces As a further example, data processing system300 may be a personal digital assistant (PDA) device, which isconfigured with ROM and/or flash ROM in order to provide non-volatilememory for storing operating system files and/or user-generated data.

The depicted example in FIG. 3 and above-described examples are notmeant to imply architectural limitations. For example, data processingsystem 300 also may be a notebook computer or hand held computer inaddition to taking the form of a PDA. Data processing system 300 alsomay be a kiosk or a Web appliance.

The mechanism of the present invention performs traces of programexecution to obtain performance and trace data for analysis. The presentinvention makes use of a plurality of counters for counting variousmetrics during the execution and tracing of the program. These countersincrement as events associated with these various metrics occur duringthe execution of the program. When the counters achieve a particularvalue, e.g., when the counters overflow, the counters generate aninterrupt which is sent to a performance monitoring tool, e.g., a traceprogram. The trace program outputs a trace record to a trace buffer ortrace file that identifies the trace information of interest for theparticular event associated with the counter that generated theinterrupt. A timer then resets the counters and execution of the programcontinues.

A post-processor application that is capable of separating out traceprofiles for different event/metric types processes the generated tracebuffer or trace file records. Thus, the post-processor application maytake a single trace buffer or trace file representation of the trace ofthe computer program that has a mixture of event/metric-based records,and generate separate trace profiles for each event/metric. In this way,a single run of the application under trace may generate trace profilesfor a plurality of events/metrics of interest. Post-processing optionsallow any metric to be chosen for roll up by subroutine, so thatpost-processing may generate arcflow reports, such as those described in“A Unifying Approach to Performance Analysis in the Java Environment”,IBM Systems Journal, Vol. 39, No. 1, 2000, pgs. 118-134, for any metric.

With reference now to FIG. 4, a block diagram depicts components used toperform performance traces of processes in a data processing system.Trace tool 400 profiles process 402, which may be a process in anapplication being traced. Trace tool 400 records data upon the executionof a hook, which is a specialized piece of code at a specific locationin a routine or program in which other routines may be connected. Tracehooks are typically inserted for the purpose of debugging, performanceanalysis, or enhancing functionality. These trace hooks send trace datato trace tool 400, which stores the trace data in buffer 404.

The trace data in buffer 404 may be subsequently stored in trace file405 or a consolidated buffer when buffer 404 is filled forpost-processing. Alternatively, the trace data may be processed inreal-time. Post-processor 406 processes the trace data located in eitherbuffer 404 or trace file 405. Post-processor 406 processes the tracedata to generate an indexed database of symbolic data for loadedmodules, as described more fully hereafter.

In a non-Java environment, trace hooks may aid in the identification ofmodules that are used in an application under trace. With Java operatingsystems, trace hooks may aid in identifying loaded classes and methods.

In addition, since a class loader may load and unload classes andmodules in a Java environment, trace data may also identify thesechanges. This is especially relevant with “network client” dataprocessing systems, such as those that may operate under Java OS, sincethe loading and unloading of classes and jitted methods may occurfrequently due to the constrained memory and role as a network client.Note that class or module load and unload information are also relevantin embedded application environments, which tend to be memoryconstrained.

With reference now to FIG. 5, a diagram depicts various phases inperforming a performance trace of the workload running on a system.Subject to memory constraints, the generated trace output may be as longand as detailed as the analyst requires for the purpose of profiling aparticular program.

An initialization phase 500 captures the state of the client machine atthe time a performance tool initiates tracing. This trace initializationdata includes trace records that identify all existing threads, allloaded classes (modules), and all methods (sections) for the loadedclasses (modules). The trace tool writes records for trace data capturedfrom hooks to a trace file or trace buffer to indicate thread switches,interrupts, and loading and unloading of classes (modules) and “jitted”methods (sections).

Any loaded class (module) has trace records that indicate the name ofthe class (module) and its methods (sections). The trace records mayalso contain numeric IDs (identifiers for threads, classes, and methods)associated with the names of the loaded classes output in the tracerecords. Trace records indicate when all of the start up information hasbeen written.

During the profiling phase 502, the trace tool writes trace records to atrace buffer or trace file. In the present invention, a trace buffer mayhave a combination of types of records, such as those that may originatefrom a trace hook executed in response to a particular type of event,e.g., when a branch is taken, and those that may originate from a stackwalking function executed in response to a timer interrupt, e.g., astack unwind record, also called a call stack record.

For example, the following operations may occur during the profilingphase if the user of the profiling utility requests sample-basedprofiling information. Each time a particular type of timer interruptoccurs, the trace tool writes a trace record, which indicates the systemprogram counter used in identifying the routine that is interrupted. Inthe depicted example, a timer interrupt initiates gathering of tracedata. Of course, other types of interrupts, such as interrupts based ona programmed performance monitor event or other types of periodicevents, may be used other than timer interrupts.

In the post-processing phase 504, the post-processing applicationprocesses the data collected in the trace buffer or sent to a trace fileor a consolidated buffer if the trace buffer is filled forpost-processing. In one configuration, the file may be sent to a server,which determines the profile for the processes on the client machine. Ofcourse, depending on available resources, a client machine may performthis post-processing step.

The present invention may be implemented with either a sample basedprofiling tool, an event-based profiling tool, or any combinationprofiling tool that uses both sample based and event based profiling. Anevent based profiling tool operates similar to the operation describedabove with regard to FIG. 4 and the use of trace hooks to generate tracerecords when a branch is taken. A sample based profiling tool operatessimilar to the operation discussed above with regard to timer basedprofiling tools that write trace records in response to timerinterrupts.

In particular, in one exemplary embodiment of the present invention, thetrace tool 400 of the present invention is implemented as an enhancedversion of the tprof trace tool available from International BusinessMachines Corporation of Armonk, N.Y. It should be appreciated that,while the preferred embodiments of the present invention may bedescribed in terms of an improved version of the tprof trace tool, thepresent invention is not limited to use with the tprof trace tool.Rather, any sample-based or event based trace tool may be augmented withthe mechanisms of the present invention in order to permit monitoring ofmultiple metrics in a single profiling run of an application, as will beapparent to those of ordinary skill in the art in view of thisdescription, without departing from the spirit and scope of the presentinvention.

The tprof trace tool is a timer profiler, which ships with the AdvancedInteractive Executive (AIX) operating system from International BusinessMachines (IBM) Corporation. This program takes samples, which areinitiated by a timer. Upon expiration of a timer, tprof identifies theinstruction executed. Tprof is a CPU trace tool for use in systemperformance analysis. The tprof tool provides a sampling techniqueencompassing the following steps: interrupt the system periodically bytime or performance monitor counter (discussed hereafter); determine theaddress of the interrupted code along with process id (pid) and threadid (tid); record a TPROF hook in the software trace buffer; and returnto the interrupted code.

In a typical use, while running an application of interest, the tproftrace tool wakes up every N milliseconds and records exactly where(e.g., at what memory address) the application is executing. If thetprof trace tool performs this step thousands of times, the tprof tracetool may generate an accurate profile of where the application isspending time, i.e. where the hotspots are, which informs those viewingthe trace information where to attempt improvements in performance ofthe application. In other uses of the tprof trace tool, the tprof toolwakes up after every Nth occurrence of a hardware event, such as a level1 data cache miss. The resulting records generated by the tprof toolidentifies which modules in the application are causing the most cachemisses and application developers may then attempt to modify theapplication to improve its performance with regard to cache misses.

It is important to note, however, that prior to the present invention,the tprof trace tool has only been able to be configured to operate inconjunction with a single metric, e.g., time, level 1 cache misses, etc.If there is a need to obtain information for a plurality of metrics, theapplication must be run repeatedly with the tprof trace toolreconfigured for each metric of interest, one metric per run of theapplication. Running the application repeatedly takes additional time ofcourse, but also, unless the application is perfectly deterministic,apparent correlations between two sets of events recorded by the tracetool cannot be made with certainty.

In a preferred embodiment, the mechanism of the present inventionprovides an improved tprof trace tool that operates to generate tracerecords associated with a plurality of metrics in a single run of theapplication being traced. This improved tprof trace tool providesmultiple counters for counting events associated with different metricsof interest (e.g., one counter associated with a number of level 1 cachemisses, a second counter associated with branch mispredictions, etc.)

The mechanism of the present invention augments the tprof trace tool toinclude functionality for associating criteria with each counter as towhen the counter will generate an interrupt resulting in the tprof tracetool outputting a trace record to a trace buffer and/or trace file. Thecriteria may be the same or different for each counter. Thus, the tproftrace tool may generate a different number of records for each type ofmetric. Moreover, the tprof trace tool may use a timer-based criteria todetermine when to generate trace records to thereby ensure the samenumber of trace records for each metric. Each of these mechanisms willbe described in greater detail hereafter.

In another exemplary embodiment of the present invention, the trace tool400 may be a sample-based or event based trace tool, such as the itracetool available from International Business Machines Corporation ofArmonk, N.Y. The present invention augments the itrace tool to includethe ability to obtain performance data for a plurality of metrics duringa single performance monitoring run of a computer program. The itracetool outputs a trace record every time the application under trace takesa branch. The trace record consists of a memory address and the numberof instructions that have been executed since the last recorded branchtaken. The mechanism of the present invention processes these tracerecords into a tree of callstacks annotated with exactly how manyinstructions are executed in each callstack configuration. The processused to generate this tree of callstacks is referred to as “arcflow.”Co-pending and commonly assigned U.S. patent application Ser. No.10/777,909 (Attorney Docket No. AUS920030825US1), entitled “Method andApparatus for Removal of Asynchronous Events in Complex ApplicationPerformance Analysis” filed on Feb. 12, 2004, and which is herebyincorporated by reference, provides examples of the itrace and arcflowtools.

It is important to note that, prior to the present invention, the itraceand arcflow tools have only recorded and processed trace records for asingle metric at a time, i.e. number of instructions. If one desiresmultiple metrics, it is necessary to run the itrace and arcflow toolsmultiple times, one for each metric of interest. This leads to the sameproblems noted above with regard to the tprof tool.

In addition, the itrace tool has an extra consideration with regard tocalibration. When the itrace tool generates a trace record, for example,a routine exit, the routine may not execute some of the instructionssince entry to that routine, but rather the itrace tool may executethese instructions. Therefore, in order to get accurate informationabout the application, it is necessary to determine how manyinstructions the itrace tool executed so they may be subtracted from thenumber of instructions itrace records as being executed since a lastbranch taken. A “number of instructions” metric determines the number ofexecuted instructions quite accurately. However, for other metrics, suchas cache misses, branch mispredictions, etc., such determinations cannotbe made easily using real hardware. The simulation environment maycalculate these other metrics accurately and thus these metrics may bequite valuable.

One exemplary embodiment of the present invention provides a mechanismthat augments the itrace tool, i.e. an event-based trace tool, such thatthe trace records generated include information for a plurality ofdifferent metrics. This improvement to the itrace tool permits theitrace tool to obtain metric information from a plurality of countersthat are configured to count events associated with a variety ofmetrics. A post-processing mechanism generates a plurality of differenttrace profiles for each metric based on the metric information stored ineach of the trace records for each of the plurality of metrics.

Turning next to FIG. 6, a block diagram of a processor system forprocessing information is depicted in accordance with a preferredembodiment of the present invention. FIG. 6 illustrates an exemplaryembodiment of the present invention in which hardware performancemonitor counters implement the counters for counting events associatedwith a plurality of metrics as in a processor 610. However, it should beappreciated that software-based counters may also implement the countersof the present invention. Moreover, the present invention may use anycombination of hardware and software counters without departing from thespirit and scope of the present invention.

Processor 610 may be implemented as processor 202 in FIG. 2. In apreferred embodiment, processor 610 is a single integrated circuitsuperscalar microprocessor. Accordingly, as discussed further hereinbelow, processor 610 includes various units, registers, buffers,memories, and other sections, all of which are formed by integratedcircuitry. Also, in the preferred embodiment, processor 610 operatesaccording to reduced instruction set computer (“RISC”) techniques. Asshown in FIG. 6, a connection exists between system bus 611 and businterface unit (“BIU”) 612 of processor 610. BIU 612 controls thetransfer of information between processor 610 and system bus 611.

A connection also exists between BIU 612, instruction cache 614, anddata cache 616 of processor 610. Instruction cache 614 outputsinstructions to sequencer unit 618. In response to such instructionsfrom instruction cache 614, sequencer unit 618 selectively outputsinstructions to other execution circuitry of processor 610.

In addition to sequencer unit 618, in the preferred embodiment, theexecution circuitry of processor 610 includes multiple execution units,namely a branch unit 620, a fixed-point unit A (“FXUA”) 622, afixed-point unit B (“FXUB”) 624, a complex fixed-point unit (“CFXU”)626, a load/store unit (“LSU”) 628, and a floating-point unit (“FPU”)630. FXUA 622, FXUB 624, CFXU 626, and LSU 628 input their sourceoperand information from general-purpose architectural registers(“GPRs”) 632 and fixed-point rename buffers 634. Moreover, FXUA 622 andFXUB 624 input a “carry bit” from a carry bit (“CA”) register 642. FXUA622, FXUB 624, CFXU 626, and LSU 628 output results (destination operandinformation) of their operations for storage at selected entries infixed-point rename buffers 634. Also, CFXU 626 inputs and outputs sourceoperand information and destination operand information to and fromspecial-purpose register processing unit (“SPR unit”) 640.

FPU 630 inputs its source operand information from floating-pointarchitectural registers (“FPRs”) 636 and floating-point rename buffers638. FPU 630 outputs results (destination operand information) of itsoperation for storage at selected entries in floating-point renamebuffers 638.

In response to a Load instruction, LSU 628 inputs information from datacache 616 and copies such information to selected ones of rename buffers634 and 638. If such information is not stored in data cache 616, thendata cache 616 inputs (through BIU 612 and system bus 611) suchinformation from a system memory 660 connected to system bus 611.Moreover, data cache 616 is able to output (through BIU 612 and systembus 611) information from data cache 616 to system memory 660 connectedto system bus 611. In response to a Store instruction, LSU 628 inputsinformation from a selected one of GPRs 632 and FPRs 636 and copies suchinformation to data cache 616.

Sequencer unit 618 inputs and outputs information to and from GPRs 632and FPRs 636. From sequencer unit 618, branch unit 620 inputsinstructions and signals indicating a present state of processor 610. Inresponse to such instructions and signals, branch unit 620 outputs (tosequencer unit 618) signals indicating suitable memory addresses storinga sequence of instructions for execution by processor 610. In responseto such signals from branch unit 620, sequencer unit 618 inputs theindicated sequence of instructions from instruction cache 614. If one ormore of the sequence of instructions is not stored in instruction cache614, then instruction cache 614 inputs (through BIU 612 and system bus611) such instructions from system memory 660 connected to system bus611.

In response to the instructions input from instruction cache 614,sequencer unit 618 selectively dispatches the instructions to selectedones of execution units 620, 622, 624, 626, 628, and 630. Each executionunit executes one or more instructions of a particular class ofinstructions. For example, FXUA 622 and FXUB 624 execute a first classof fixed-point mathematical operations on source operands, such asaddition, subtraction, ANDing, ORing and XORing. CFXU 626 executes asecond class of fixed-point operations on source operands, such asfixed-point multiplication and division. FPU 630 executes floating-pointoperations on source operands, such as floating-point multiplication anddivision.

As information is stored at a selected one of rename buffers 634, suchinformation is associated with a storage location (e.g. one of GPRs 632or CA register 642) as specified by the instruction for which theselected rename buffer is allocated. Sequencer unit 618 copiesinformation stored at a selected one of rename buffers 634 to itsassociated one of GPRs 632 (or CA register 642). Sequencer unit 618directs such copying of information stored at a selected one of renamebuffers 634 in response to “completing” the instruction that generatedthe information. Such copying is called “writeback.”

As sequencer unit 618 copies information from one of rename buffers 638,such information is associated with one of FPRs 636. Sequencer unit 618copies information stored at a selected one of rename buffers 638 to itsassociated one of FPRs 636 in response to signals from sequencer unit618. Sequencer unit 618 directs such copying of information stored at aselected one of rename buffers 638 in response to “completing” theinstruction that generated the information.

Processor 610 achieves high performance by processing multipleinstructions simultaneously at various ones of execution units 620, 622,624, 626, 628, and 630. Accordingly, processing of each instructionoccurs as a sequence of stages, each being executable in parallel withstages of other instructions. Such a technique is called “pipelining.”In a significant aspect of the illustrative embodiment, processing of aninstruction occurs as six stages, namely fetch, decode, dispatch,execute, completion, and writeback.

In the fetch stage, sequencer unit 618 selectively inputs (frominstruction cache 614) one or more instructions from one or more memoryaddresses storing the sequence of instructions discussed furtherhereinabove in connection with branch unit 620, and sequencer unit 618.

In the decode stage, sequencer unit 618 decodes up to four fetchedinstructions. In the dispatch stage, sequencer unit 618 selectivelydispatches up to four decoded instructions to selected (in response tothe decoding in the decode stage) ones of execution units 620, 622, 624,626, 628, and 630 after reserving rename buffer entries for thedispatched instructions' results (destination operand information). Inthe dispatch stage, operand information is supplied to the selectedexecution units for dispatched instructions. Processor 610 dispatchesinstructions in order of their programmed sequence.

In the execute stage, execution units execute their dispatchedinstructions and output results (destination operand information) oftheir operations for storage at selected entries in rename buffers 634and rename buffers 638 as discussed further hereinabove. In this manner,processor 610 is able to execute instructions out-of-order relative totheir programmed sequence.

In the completion stage, sequencer unit 618 indicates an instruction is“complete.” Processor 610 “completes” instructions in order of theirprogrammed sequence.

In the writeback stage, sequencer 618 directs the copying of informationfrom rename buffers 634 and 638 to GPRs 632 and FPRs 636, respectively.Sequencer unit 618 directs such copying of information stored at aselected rename buffer. Likewise, in the writeback stage of a particularinstruction, processor 610 updates its architectural states in responseto the particular instruction. Processor 610 processes the respective“writeback” stages of instructions in order of their programmedsequence. Processor 610 advantageously merges an instruction'scompletion stage and writeback stage in specified situations.

In the illustrative embodiment, each instruction requires one machinecycle to complete each of the stages of instruction processing.Nevertheless, some instructions (e.g., complex fixed-point instructionsexecuted by CFXU 626) may require more than one cycle. Accordingly, avariable delay may occur between a particular instruction's executionand completion stages in response to the variation in time required forcompletion of preceding instructions.

Sequencer unit 618 includes a completion buffer 648 to track thecompletion of the multiple instructions which are being executed withinthe execution units. Upon an indication that an instruction or a groupof instructions have been completed successfully, in an applicationspecified sequential order, completion buffer 648 initiates the transferof the results of those completed instructions to the associatedgeneral-purpose registers.

In addition, processor 610 also includes processor monitoring unit 64connected to instruction cache 614, as well as other units in processor610. Performance monitor unit 640, which in this illustrative embodimentis a software-accessible mechanism capable of providing detailedinformation descriptive of the utilization of instruction executionresources and storage control, monitors the operation of processor 610.Although not illustrated in FIG. 6, performance monitor unit 640 iscoupled to each functional unit of processor 610 to permit themonitoring of all aspects of the operation of processor 610, including,for example, reconstructing the relationship between events, identifyingfalse triggering, identifying performance bottlenecks, monitoringpipeline stalls, monitoring idle processor cycles, determining dispatchefficiency, determining branch efficiency, determining the performancepenalty of misaligned data accesses, identifying the frequency ofexecution of serialization instructions, identifying inhibitedinterrupts, and determining performance efficiency.

Performance monitor unit 640 includes an implementation-dependent number(e.g., 2-8) of counters 641-642, labeled PMC1 and PMC2, which areutilized to count occurrences of selected events. Performance monitorunit 640 further includes at least one monitor mode control register(MMCR). In this example, two control registers, MMCRs 643 and 644specify the function of counters 641-642. Counters 641-642 and MMCRs643-644 are preferably implemented as SPRs that are accessible for reador write via MFSPR (move from SPR) and MTSPR (move to SPR) instructionsexecutable by CFXU 626. However, in one alternative embodiment, counters641-642 and MMCRs 643-644 may be implemented simply as addresses in I/Ospace. In another alternative embodiment, access to the controlregisters and counters occurs indirectly via an index register. Thisembodiment may be implemented, for example, in the IA-64 architecture inprocessors from Intel Corporation.

Additionally, processor 610 also includes interrupt unit 650, connectedto instruction cache 614. Additionally, although not shown in FIG. 6,connections exists between interrupt unit 650 and other functional unitswithin processor 610. Interrupt unit 650 receives signals from otherfunctional units and initiates an action, such as starting an errorhandling or trap process. In these examples, interrupt unit 650generates interrupts and exceptions that may occur during execution of aprogram.

Co-pending and commonly assigned U.S. patent application Ser. No.10/757,256 (Attorney Docket No. AUS920030545US1), entitled “Method andApparatus for Autonomic Dectection of ‘Chase Tail’ Conditions andStorage of Instruct/Data in ‘Chase Tail’ Data Structure,” filed on Jan.14, 2004, which is hereby incorporated by reference, provides exemplaryoperations and uses of performance monitor counters 641-642 and theother elements shown in FIG. 6.

As mentioned above, the present invention improves upon known tracetools by providing functionality for utilizing a plurality of countersthat count events associated with a plurality of metrics of interestduring a single run of the application being traced. For example, thepresent invention provides functionality for obtaining count informationfrom performance monitor counters 641-642 in response to the meeting ofcertain criteria, i.e. either time criteria or other types of criteria,during time-based or event-based profiling of the application.

During the tracing of the run of the application, performance monitorcounters 641-642 count events associated with their respective metrics.The present invention provides trace tools with functionality such thatin response to the occurrence of particular criteria associated with thecounters, the trace tools output a trace record for the particular eventtype associated with the counter that meets the criteria. Alternatively,in response to the occurrence of particular criteria, the trace toolprovides for obtaining the current counts of the various performancemonitor counters, such as 641-642, and writing the counts to a tracerecord. A timer reinitializes the performance monitor counters 641-642with counts meeting the criteria or obtained for writing to the tracerecord, and the process repeats.

As mentioned above, rather than implementing performance monitorcounters 641-642 in hardware as depicted in FIG. 6, the presentinvention may make use of software-based counters or a combination ofhardware and software-based counters. With regard to software-basedcounters, such counters may be, for example, in one or more interrupthandlers associated with the hooks placed in the application beingtraced. Thus, when an interrupt occurs in response to encountering atrace hook during execution of the application, a correspondinginterrupt handler processes the interrupt. A counter associated with theinterrupt handler increments for the particular event associated withthe particular metric corresponding to the encountered trace hook. Thetrace tool may then obtain the counts of the various counters associatedwith the one or more interrupt handlers to thereby generate a tracerecord with the counters being reinitialized thereafter.

The criteria utilized for determining when to generate a trace recordbased on counts of events associated with a plurality of metrics maytake a variety of forms. In one exemplary embodiment of the presentinvention, the trace tool samples several different performance monitorcounters (either hardware-based or software-based) every N_(i) events,where i is a performance monitor counter identifier and the N's can bedifferent for each performance monitor counter. Each counter is set toits an initial value based on its capacity and the number of eventsN_(i). For example, the counter may be set to an initial value based onthe following equation:capacity−N_(i)+1

After N_(i) events of this type, e.g., number of instructions executed,number of cache misses, number of branch mispredictions, etc., thecounter i overflows and generates an interrupt. In response to thegeneration of this interrupt, the trace tool generates a trace record.The trace record may include, for example, an event type identifier, atimestamp, a memory address associated with the instruction currentlybeing executed by the application at the time of the interrupt, and thelike.

Following generation of the trace record, the mechanism of the presentinvention resets the counter i to its initialized value and execution ofthe application under trace resumes. Using this methodology, the tracetool may generate a different number of trace records for each type ofevent associated with each metric of interest. The trace tool generatesa number of trace records for a particular type of event and metricbased on a combination of the frequency of that event type occurring andthe value of N_(i) chosen for that particular type of event.

In another exemplary embodiment, the mechanism of the present inventionemploys a timer for adjusting the values of the counters such that theywill generate an interrupt on a next occurrence of an associated eventtype. In this alternative embodiment, a timer may be set for apredetermined interval at which the code will initiate resetting of thecounters, or a subset of the counters, to their capacity. After thetimer goes off and the counters are reset to their capacity, the timerresets.

On the next event of each type associated with the counters that havebeen set to their capacity, the associated counter will overflow andgenerate an interrupt. At this point, the trace tool outputs a tracerecord. Again, this trace record may include, for example, an event typeidentifier, a timestamp, a memory address associated with theinstruction currently being executed by the application at the time ofthe interrupt, and the like.

Following generation of the trace record, the mechanism of the presentinvention resets the counter that initiated the generation of the tracerecord to its initialized value and execution of the application undertrace resumes. Using this methodology, the trace tool generates the samenumber of trace records for each event type/metric regardless of theirnaturally occurring frequency in general. However, in other embodimentsin which this methodology is combined with the previously describedmethodology, frequency of occurrence of particular events may beimportant for those metrics whose events have frequencies such thatN_(i) is met prior to the timer interval being met.

For both methodologies described above, trace records that are outputmay be identical. Thus, the same trace tool may use both methodologiestogether or interchangeably depending on the desired operation of thetrace tool. In this way, the same post-processing may be performed ontrace records generated by either methodology. This post-processinginvolves traversing the trace records generated by the trace tool,identifying the various event types, and generating separate profilesfor each event type. In other words, the post-processing comprises firsttraversing the trace records to separate out the trace records into aplurality of sets of trace records based on event type. Then thepost-processing generates a trace profile for the particular metricassociated with that set of trace records based on each set of tracerecords in the plurality of sets of trace records. The post-processingmechanism then generates one or more reports comprising these varioustrace profiles for use by a human user.

Thus, for example, the post-processing mechanism generates a profileshowing which modules consumed the most processor time, a profileshowing which modules cause the most cache misses, etc. The generationof profiles from trace records is generally known in the art. Animprovement with the post-processing performed by the present inventionis the ability to discern the various event types to thereby generateseparate profiles from trace records generated in a single trace of asingle execution of an application. A single trace file, trace buffer,etc., may store these trace records. In addition, the post-processingmechanism may generate ratios, e.g., cache misses per second, in eachmodule provided care is taken to account for the frequency of theoccurrence of the events versus the chosen sampling rate.

The above methodology may be implemented, for example, in associationwith the tprof trace tool. In addition, other mechanisms for generatinga trace record based on a counter, from a plurality of counters, havinga value that meets a predetermined criteria may be used withoutdeparting from the spirit and scope of the present invention.

In yet another exemplary embodiment of the present invention, the tracetool may generate trace records having information about the change incounter values for each of the plurality of counters associated withdifferent metrics, at the occurrence of each branch taken. For allchosen metrics, initial value of a corresponding counter (eitherhardware or software counter) at the start of tracing is stored astracing of the application may be turned on and off independent of thestart and termination of the application being traced.

Thereafter, at each branch taken, the mechanism of the present inventioncalculates a change in the values of each counter since a lastobservation. For each branch taken, the trace tool outputs a tracerecord in which this record contains all the metrics for the branchtaken, including, for example, counters identifying cache misses, level2 interventions, and TLB misses. The translation lookaside buffer (TLB)is a table in the processor that contains cross-references between thevirtual and real addresses of recently referenced pages of memory. Itfunctions like a “hot list” or quick-lookup index of the pages in mainmemory that have been most recently accessed.

When a cache miss occurs, this quick-lookup index allows for fetchingdata from an address in virtual memory. If the real-memory address of adesired page is not in the TLB, the real address must be determined byother means, thus causing a further delay. “TLB space” is conceptuallythe amount of accessible memory by looking up an address quickly in theTLB. The TLB it is not a physically distinct area of memory, but ineffect, it does lead to a “faster” subset of main memory. TLB space istherefore equal to the number of addresses stored in the TLB times thepage size. Page size is 4096 bytes or 4K. For a Pentium III processorwith 64 addresses in the TLB, the TLB size is 64×4K or 256K.

The trace record may include, for example, an event type identifier, achange in the counter value, a memory address, and the like. Thus, ifthere are eight different counters counting events associated with eightdifferent metrics, then each time a branch is taken, the trace tooloutputs eight trace records identifying the change in the correspondingcounter value since a last observation of the counter value.

The post-processing of trace records generated using this methodologyinvolves constructing a separate callstack tree for each chosen metric.The post-processing operation first traverses the trace records toseparate out trace records based on event type into a plurality of setsof trace records. Thereafter, the post-processing operation generates aseparate callstack tree for the associated metric for each set of tracerecords in the plurality of sets of trace records. Co-pending andcommonly assigned U.S. patent application Ser. No. 10/777,909 (AttorneyDocket No. AUS920030825US1), which is hereby incorporated by reference,provides an example of the generation of callstack trees from tracerecords.

Because a trace record is output for each metric at each branch taken,even if the change in the value of the counter associated with thatmetric is zero, the callstack trees for the various metrics will beidentical in shape. This makes comparisons of callstack trees andgeneration or ratios among the metrics straightforward.

This embodiment of the present invention may be used, for example, inassociation with the itrace tool described previously. As mentionedabove, the itrace tool generates a trace record each time a branch istaken. The mechanism of the present invention augments the itrace toolto generate a plurality of trace records at each branch taken with eachtrace record corresponding to a particular performance monitor counterassociated with a metric of interest. In addition, the mechanism of thepresent invention allows for modifying the trace to identify the eventtype associated with the corresponding counter and the change in thecounter value since a last observation of the counter value.

Regardless of which particular embodiment is utilized by animplementation of the present invention, it is necessary to performsymbolic resolution of the memory addresses stored in the trace recordsso as to provide a meaningful output to a user. Symbolic resolutiontranslates addresses into symbolic names that have meaning to humanusers. Any symbolic resolution methodology may be used by the variousembodiments of the present invention without departing from the spiritand scope of the present invention. Commonly assigned U.S. Pat. No.6,766,511, entitled “Apparatus and Method for Performing SymbolicResolution of Modules Using Static Representations of a Trace,” issuedto Berry et al. on Jul. 20, 2004, which is hereby incorporated byreference, provides one exemplary symbolic resolutionmechanism/methodology. Alternatively, the symbolic resolution may beperformed using a merged symbol file as described, for example, incommonly assigned and co-pending U.S. patent application Ser. No.09/613,190 (Attorney Docket No. AUS000127US1), entitled “Apparatus andMethod for Cataloging Symbolic Data for Use in Performance Analysis ofComputer Programs,” filed on Jul. 10, 2000, which is hereby incorporatedby reference.

With reference now to FIG. 7, a diagram illustrating primary operationalcomponents according to one exemplary embodiment of the presentinvention is depicted. During the initialization phase of a performancetracing program, such as initialization phase 500 in FIG. 5, trace tool720 may instruct operating system 740 regarding the metrics of interestduring the trace of the application. Device driver 742 of operatingsystem 740 may communicate with microcode 712 of processor 710 to setthe values of control registers 702 to identify the metrics of interestand the counters associated with those metrics. For example, devicedriver 742 may instruct microcode 712 to set the values in MMCRs 718 toidentify the various event types associated with corresponding PMCs 716.PMCs 716 may count the occurrence of events up to a maximum countervalue. When the counter reaches a maximum value and another event ofthat type occurs, the counter will overflow thereby generating aninterrupt to operating system 740.

In some embodiments, the mechanism of the present invention providestimer 714 with a corresponding time interval. Timer 714 increments withevery time step until reaching a time interval, at which time the timerwill generate an interrupt to operating system 740.

After initialization of MMCRs 718, PMCs 716, and timer 714, executionand tracing of application 730 may commence. As application 730 isexecuting and instructions of application 730 are executed by processor710, microcode 712 identifies various event types and incrementscorresponding PMCs 716. When a PMC 716 or timer 714 overflows, thisgenerates an interrupt, which is sent to operating system 740. Operatingsystem 740 calls a corresponding interrupt handler 780 associated withtrace application 720.

Interrupt handler 780 may cause one or more trace records to begenerated in response to the interrupt. In an alternative embodiment, asdescribed above, interrupt handler 780 may cause the values of PMCs 716to be reset to their capacity. The mechanism of the present inventionstores the trace records in trace file 722 and/or trace buffer 724. Thetrace record corresponds to a particular event type associated with acounter. As mentioned above, depending on the particular embodimentbeing implemented, the trace tool may generate a trace record for theevent type corresponding to the counter that generated the interrupt orfor all of the event types and counters.

Post-processing program 728 processes the trace records in trace file722 and/or trace buffer 724 to generate separate profiles for each eventtype corresponding to the metrics of interest. This post-processing mayinvolve traversing each trace record to generate sets of trace recordsbased on event type. Thereafter, the post-processing uses each set oftrace records to generate profiles and/or callstack trees for output.This process may also use symbolic data to perform address to nametranslation, i.e. symbolic resolution, such as symbolic resolutionperformed using merged symbol file 735.

In the special case when the trace tool generates a trace on a simulatedmachine, while it is possible to execute post-processing program 728 onthe same simulated machine, it is not desirable to do so becausesimulation is typically orders of magnitude slower than real hardware.To accomplish post processing faster, we can transfer trace file 722 toa separate, real machine and execute post-processing program 728 there.However, in some real environments, accomplishing the first steps inpost processing, namely creating merge symbol file 735 and indexedsymbolic database 726 on a different platform from where the trace wasgenerated, is difficult. This is because in some environments, forexample in the Linux operating system, trace file 722 contains the namesof modules and the address at which a module was loaded, but not theaddresses of symbols within the module (for example, the trace containsthe address of a program, but not the addresses of subroutines withinthe program). In these environments, post-processing program 728 looksinside the module where names of individual routines and theircorresponding address may be found. If a different machine performs thepost-processing operation, the module may not exist, or worse, adifferent module of the same name may be present.

Anticipating this difficulty, while creating trace file 722 on asimulated machine, trace tool 720 adds an extra directory prefix to thename of all modules it encounters. For example, if the name of a moduleis /usr/bin/program_name (module names are usually recorded with theirfull file system pathname), the trace tool records its address and thename /simtree/usr/bin/program_name in the trace file. Later, beforebeginning to post process on a real machine, we create a new rootdirectory /simtree, and copy or mount the disk image from the simulatedenvironment to this new directory. Then when the name/simtree/usr/bin/program_name is encountered in the local copy of tracefile 722, post-processing program 728 can look in local directory/simtree/usr/bin, find a binary image of program_name, and look insideit to find internal symbols.

U.S. patent application Ser. No. ______ (Attorney Docket No.AUS9000127US1), “Apparatus and Method for Cataloging Symbolic Data forUse in Performance Analysis of Computer Programs”, incorporated byreference above, provides a method for post-processing performance tracedata recorded in trace buffer 724 or trace file 722. The post-processinggenerates a merged symbol file, such as merge symbol file 735, for acomputer program, or application, under trace. Merged symbol file 735comprises symbolic data for modules obtained from map files, debugfiles, non-stripped versions of modules, and other symbolic data files.The merged symbol file contains information useful in performingsymbolic resolution of address information in trace files for eachinstance of a module, such as checksum, timestamp, fully qualified pathto the module, length of the module, etc.

During post processing of the trace information generated by aperformance trace of a computer program, the post-processing operationcompares the symbolic information stored in merged symbol file 735 tothe trace information stored in trace file 722. The trace informationincludes information identifying the modules that were called during thetrace of the computer application. The post-processing operation usesthis trace information, which may be obtained using the hardware threadtracking mechanisms previously described, and the merged symbol file toproduce reports. The correct symbolic information in merged symbol file735 for the modules used in the trace is identified based on a number ofvalidating criteria.

The post-processing operation may then store the correct symbolicinformation for the required modules as an indexed symbolic database,for example, indexed symbolic database 726, which is indexed usingprocess identifiers and address identifiers. The post-processingoperation may store the indexed database of symbolic information as aseparate file or as a separate portion of a trace file for the computerapplication. The post-processing operation may then use the indexedsymbolic database 726 to resolve address information into correspondingsymbolic information when providing the trace information for use by auser.

As described above, the symbolic information provides symbolic data forloaded modules/processes, i.e. called module or processes, of theapplication under trace. As a result of the symbolic resolution, eithertrace tool 720 or post-processing program 728 generates indexed symbolicdatabase 726 for the loaded/called modules/processes. The indexeddatabase entries may be indexed based on any searchable value. In apreferred embodiment, the indexed database is indexed based on theprocess identifier (pid) and the segment load address, however, othersearchable indices may be used without departing from the spirit andscope of the present invention.

Once the post-processing operation generates indexed symbolic database726, trace tool 720 or post-processing application 728 may searchindexed symbolic database 726 for symbolic information that matches theprocess identifier (pid) and the address of the method/routine called orreturned by each thread. When a match is found, the present inventiongenerates a call sequence using symbolic data that represents thecomputer program flow.

The trace records in trace file 722 and/or trace buffer 724 may takedifferent forms depending on the particular embodiments of the presentinvention implemented. FIG. 8A illustrates an example of the tracerecords generated for exemplary embodiments of the present invention inwhich the trace record is generated in response to a counter overflowgenerating an interrupt.

As shown in FIG. 8A, the trace record includes event type 810corresponding to the event types counted by the corresponding counter,timestamp 820 for the occurrence of the interrupt causing the generationof the trace record, and memory address 830 for the instruction executedby the application being traced when the interrupt was generated. Thepresent invention uses event type 810 to separate trace records intosets of trace records based on event type. Post-processing mechanismsuse timestamp 820 to generate trace profiles, as is generally known inthe art. Memory address 830 is the basis for symbolic resolution withregard to this trace record.

FIG. 8B illustrates an exemplary trace record for an embodiment of thepresent invention in which trace records are generated responsive to aninterrupt. As shown in FIG. 8B, the trace records include event type840, change value 850 that identifies a change in counter value for thecounter corresponding to the trace record, and memory address 860. Eventtype identifier 840 and memory address 860 are similar to the tracerecord shown in FIG. 8A. Change value 850 is the difference between acurrent value of the counter and a previous value of the counter in aprevious trace record for that counter or a stored initial value of thecounter. As discussed previously, with this embodiment, the trace toolwrites a separate trace record for each counter when an interruptoccurs.

FIG. 8C illustrates an exemplary trace record for an embodiment of thepresent invention in which a trace tool generates trace records forevery branch taken. As shown in FIG. 8C, the trace records includememory address 880 that identifies the address from which the branch istaken, memory address 882 that identifies the branch-to address, andchange values 884-888 that identify changes in counter values for thecounter corresponding to the trace record. Change values 884-888 are thedifferences between a current value of the counter and a previous valueof the counter in a previous trace record for that counter or a storedinitial value of the counter. Alternative implementations may write oneor more separate trace records that indicate the metrics that havechanged. Thus, with the mechanisms of the present invention, trace toolsgenerate trace records for a plurality of different metrics during asingle run of an application and write these trace records to a singletrace file and/or trace buffer, which are later extracted out intoseparate sets of trace records for use in generating separate traceprofiles or callstack trees for each metric of interest. Thus, themechanism of the present invention allows for generating multiple traceprofiles and callstack trees for multiple metrics using only a singlerun of an application.

For situations in which application runs may be require an extensiveamount of time, such as with system simulation, the present inventionresults in a great reduction in run time to obtain all of theperformance information required for the various metrics of interest.That is, for example, with a complex computer program capable ofsimulating all hardware aspects of an entire computer system, includingdetails of processor units, caches, buses, memory, multiprocessors,etc., a run of such a computer program may require many days tocomplete. With conventional tracing tools, multiple runs of this complexcomputer program would be required in order to obtain all of theperformance information necessary to properly evaluate the operation ofthe application. With the present invention, only a single run of thecomputer program is necessary in order to obtain all of the requiredperformance information for the plurality of metrics of interest.

FIGS. 9-11 are flowcharts outlining exemplary operations of exemplaryembodiments of the present invention. It will be understood that eachblock of the flowchart illustrations, and combinations of blocks in theflowchart illustrations, can be implemented by computer programinstructions. These computer program instructions may be provided to aprocessor or other programmable data processing apparatus to produce amachine, such that the instructions which execute on the processor orother programmable data processing apparatus create means forimplementing the functions specified in the flowchart block or blocks.These computer program instructions may also be stored in acomputer-readable memory or storage medium that can direct a processoror other programmable data processing apparatus to function in aparticular manner, such that the instructions stored in thecomputer-readable memory or storage medium produce an article ofmanufacture including instruction means which implement the functionsspecified in the flowchart block or blocks.

Accordingly, blocks of the flowchart illustrations support combinationsof means for performing the specified functions, combinations of stepsfor performing the specified functions and program instruction means forperforming the specified functions. It will also be understood that eachblock of the flowchart illustrations, and combinations of blocks in theflowchart illustrations, can be implemented by special purposehardware-based computer systems which perform the specified functions orsteps, or by combinations of special purpose hardware and computerinstructions.

FIG. 9 is a flowchart outlining an exemplary operation of the presentinvention in which a trace tool generates trace records in response to acounter generated interrupt. As shown in FIG. 9, the operation starts byinitializing the counters for the various metrics of interest (step910). As mentioned above this initialization, in one exemplaryembodiment, may involve setting the counter initial values a value basedon their capacity and an interval number of events between tracerecords.

Thereafter, the application runs with counters counting events duringthe running of the application (step 920). A determination is made as towhether a counter has overflowed and generated an interrupt (step 930).If so, the interrupt is routed to an appropriate interrupt handler (step940) that generates a trace record identifying the event type, thetimestamp and memory address corresponding to the interrupt (step 950).The counter generating the interrupt is then reinitialized (step 960)and a determination is made as to whether a termination event occurred(step 970), e.g., termination of the trace. If a termination event hasnot occurred, the operation returns to step 920. Otherwise, theoperation ends.

FIG. 10 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention in which counter values are set totheir capacity in response to a timer interrupt. As shown in FIG. 10,the operation starts by initializing the counters to count eventsassociated with metrics of interest (step 1010). A timer is initialized(step 1020) and the application runs (step 1030) with the counterscounting events as they occur.

A determination is made as to whether a termination event has occurred(step 1035), e.g., termination of the trace. If a termination eventoccurred, then the operation terminates. Otherwise, a determination ismade as to whether a counter generated a counter interrupt due to anoverflow of the counter based on the occurrence of events during therunning of the application (step 1040). If the counter interrupt exists,the interrupt is routed to an appropriate interrupt handler whichgenerates a trace record (step 1050). Thereafter, or if the counterinterrupt does not exist, a determination is made as to whether thetimer reached a predetermined time interval (step 1060). If not, theoperation returns to step 1030 and continues to run the application.

If the timer reached a predetermined time interval, an interrupt isgenerated (step 1070). The interrupt is routed to an appropriateinterrupt handler which then sets the values of all of the counters totheir capacity (step 1080). The timer resets (step 1090) and theapplication continues to run (step 1095). The operation then returns tostep 1030.

FIG. 11 is a flowchart outlining an exemplary operation of one exemplaryembodiment of the present invention when generating trace records foreach counter at every branch taken in an application under trace. Asshown in FIG. 11, the operation starts by initializing counters forcounting events associated with metrics of interest (step 1110).Thereafter, the application starts to run with the counters incrementingin response to the occurrence of their corresponding events (step 1120).A determination is made as to whether a termination event occurred (step1130). If a termination event has occurred, the operation terminates.Otherwise, a determination is made as to whether a branch has been taken(step 1140).

If a branch has been taken, the trace tool generates a trace record foreach counter (step 1150). If a branch has not been taken, the operationreturns to step 1120.

Thus, the present invention provides an improved mechanism for tracingof applications that permits multiple metrics to be monitored andperformance data collected for a plurality of metrics of interest duringa single run of an application. The present invention providesimprovements to known time-based and event-based trace tools, such astprof and itrace, that permit these tools to be used with large andcomplex computer programs without requiring multiple runs of thesecomplex computer programs thereby reducing costs in validation,debugging, and the like.

It is important to note that while the present invention has beendescribed in the context of a fully functioning data processing system,those of ordinary skill in the art will appreciate that the processes ofthe present invention are capable of being distributed in the form of acomputer readable medium of instructions and a variety of forms and thatthe present invention applies equally regardless of the particular typeof signal bearing media actually used to carry out the distribution.Examples of computer readable media include recordable-type media, suchas a floppy disk, a hard disk drive, a RAM, CD-ROMs, DVD-ROMs, andtransmission-type media, such as digital and analog communicationslinks, wired or wireless communications links using transmission forms,such as, for example, radio frequency and light wave transmissions. Thecomputer readable media may take the form of coded formats that aredecoded for actual use in a particular data processing system.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A method in a data processing system for profiling program execution,the method comprising: initializing a plurality of counters to countevents associated with metrics of interest; counting occurrences ofevents associated with metrics of interest during a single execution ofa computer program; responsive to a determination that a counter in aplurality of counters has generated an interrupt, rerouting theinterrupt to an interrupt handler, wherein the interrupt handlergenerates trace records comprising trace information corresponding tothe interrupt; and generating profiles for a trace, wherein the profilesdifferentiate the trace records based on a metric type associated witheach trace record.
 2. The method of claim 1, further comprising:reinitializing the counter that generated the interrupt.
 3. The methodof claim 1, wherein the initializing step includes setting initialvalues of the counters to a value based on each counter's capacity. 4.The method of claim 1, wherein the interrupt is generated when a maximumvalue of the counter is reached.
 5. The method of claim 1, whereincounter values for the plurality of counters are reset to a maximumvalue when a particular time interval is reached.
 6. The method of claim1, wherein the trace record identifies an event type associated with thecounter that generated the interrupt.
 7. The method of claim 1, whereinthe generating step is performed by a timer based profiling tool.
 8. Themethod of claim 7, wherein one or more counters are set to overflow andgenerate an interrupt.
 9. The method of claim 7, wherein the timer basedprofiling tool is tprof trace tool.
 10. The method of claim 1, whereinthe metric type includes L1 cache misses, cycles, instructionscompleted, TLB misses, or any other event supported by a performancemonitor unit.
 11. The method of claim 1, wherein the metric typeincludes counts of executions of selected modules, counts of pagefaults, specific file accesses, or any other software generated event.12. The method of claim 1, wherein the generating step includes using anadditional directory name to distinguish a first name space on asimulated system being profiled from a second name space on a system onwhich the trace is reduced.
 13. A data processing system for profilingprogram execution, comprising: initializing means for initializing aplurality of counters to count events associated with metrics ofinterest; counting means for counting occurrences of events associatedwith metrics of interest during a single execution of a computerprogram; rerouting means for rerouting the interrupt to an interrupthandler in response to a determination that a counter in a plurality ofcounters has generated an interrupt, wherein the interrupt handlergenerates trace records comprising trace information corresponding tothe interrupt; and generating means for generating profiles for a trace,wherein the profiles differentiate the trace records based on a metrictype associated with each trace record.
 14. The data processing systemof claim 13, further comprising: reinitializing means for reinitializingthe counter that generated the interrupt.
 15. The data processing systemof claim 13, wherein the initializing means includes setting initialvalues of the counters to a value based on each counter's capacity. 16.The data processing system of claim 13, wherein the interrupt isgenerated when a maximum value of the counter is reached.
 17. The dataprocessing system of claim 13, wherein counter values for the pluralityof counters are reset to a maximum value when a particular time intervalis reached.
 18. The data processing system of claim 13, wherein thetrace record identifies an event type associated with the counter thatgenerated the interrupt.
 19. The data processing system of claim 13,wherein the generating step includes using an additional directory nameto distinguish a first name space on a simulated system being profiledfrom a second name space on a system on which the trace is reduced. 20.A computer program product in a computer readable medium for profilingprogram execution, comprising: first instructions for initializing aplurality of counters to count events associated with metrics ofinterest; second instructions for counting occurrences of eventsassociated with metrics of interest during a single execution of acomputer program; third instructions for rerouting the interrupt to aninterrupt handler in response to a determination that a counter in aplurality of counters has generated an interrupt, wherein the interrupthandler generates trace records comprising trace informationcorresponding to the interrupt; and fourth instructions for generatingprofiles for a trace, wherein the profiles differentiate the tracerecords based on a metric type associated with each trace record.